Chama TODO 9/6/2023:

• Rainfall
  • Make sure the format is correct to send to Anne
  • Look at the duplicates.
    • Check to see if the whole row is being duplicated if it is we can likely delete the row.
  • Find a quick test time I can create a rainfall file from and start coding the INP file generation for rainfall and for the Start and end time. This is done format works but duplicates cause an issue. Fixing this now
    • While I am working on the INP file Script I should also pull out the hard coded values from my test script. Also get Jake to give it a code review after it is done.
  • Send out email on issues with the ASOS rainfall data.
• Naming the INP file
  • Send Email on how the naming should be done from my notes.
  • Start a diagram for file structure in mermaid for the HDF5 file
    • What file will go in the HDF5 file
    • What will be compressed.
      • Think through signal files.
      • Think through INP files
      • Think through references Datasets.
  • Write up a document on HDF5 files for dev and also for Chama OneNote.
    • We have previously just read from the HDF5 files now We will need to write to them as well.
• Updated Tasklist
  • Start Documentation on HDF5 file where each signal file will have it's own HDF5 file.
  • Write up what API's would be needed to modify the SWMM Runs without generating an INP file for each File.
  • Start looking at Parquet files to store HDF5 files.
• Ensemble
  • Start a plan for actually creating the INP files.
    • Start with changing the Degrees and Start Locations of the Plume.
    • How will we be generating the files?
      • Are we going to randomize it?
      • Are we going to have a certain number of good locations to pick from to start the Plume.
      • How much time should the User spend on Setting things up?

Rainfall

1. We are using ASOS data.
   1. There is a hourly cumulative data at :54 min. The issue is that sometimes the hourly average does not line up with the other readings between the time period.
   2. Are we keeping the data?
   3. What about the data that is

Duplicates

I wanted to know if the duplicates for date time were showing the same data as in there was a double entry at that time or if there were 2 entries in a minute for some reason like High intensity rainfall.

To do this I ran two pandas commands looking for duplicates 1 was on datetime and the other was for the whole row.

There are 1742 rows that have the datetime duplicated or are duplicates.

When comparing the whole row there was 1052 lines that were duplicated or were duplicates. Therefore These duplicates can be eliminated.

Leaving Only 690 values.

I am going to check for Zeros. Lastly I am going to take the biggest of the values and let that be the value for that datetime.

Adding Rainfall to file Generation.

Functional Specifications

Sections that need to be modified or replaced.

1. OPTIONS
2. RAINGAGES
3. SUBCATCHMENTS
4. TIMESERIES

OPTIONS

fields that need modified are START_DATE START_TIME REPORT_START_DATE REPORT_START_TIME END_DATE END_TIME

RAINGAGES

CVG will need to be added as a rain gage.

SUBCATCHMENTS

All the Subcatchments's rain gage needs to be changed to CVG.

TIMESERIES

The rainfall data from the rainfall file will need to be placed in the TIMESERIES section.

If we want to make the INP files more readable we should replace the data in the RAINGAGES and TIMESERIES sections so additional rain gages are not available.

Design Specifications

Read the Rainfall File it should be placed in a pandas Dataframe called df_rain.

In this file there should be a station column which should be CVG a DateTime and a Rainfall.

Get the Value of The 1st station which should be CVG this and save it as a station_name.

Get the Min and Max date time for the Rainfall. Parse this string for Start_Date and Start_Time and End_Date and End_Time.

Now start on the sections of the INP file.

Print the comments to the INP file. A comment starts with a ';'. It makes it easier for whoever needs to read it later. Note Formatting in INP file can be either tabs or spaces.

To Navigate the INP file we will use the words in [] The only pace brackets are used in an INP file is for section names therefore we can start with a Section name in Brackets and continue until we reach the next open bracket '['

If the section starts with [RAINGAGES] Then Replace it with the following:

;;Name Format Interval SCF Source ;;-------------- --------- ------ ------ ---------- ;CVG CVG VOLUME 0:05 1.0 TIMESERIES CVG

If the section starts with [TIMESERIES]. The replace it with the pandas Dataframe df_rain. It should already be in the correct format. If the pandas dataframe prints the column headers then we will need to add a ; before the headers.

Both of the sections described are the easier sections as we want to replace the entire section.

If the section starts with [SUBCATCHMENTS]. I need to scrape the subcatchment section into a pandas dataframe called df_subcatchment. If there are any lines that start with ; they need to be written back to the file. Then I will need to set the column for the Raingage (This should be the second column of the pandas dataframe) to the station_name. Then I will replace the section with the Dataframe.

If the section starts with [OPTIONS]. The options section will also need to be scraped into a pandas dataframe called df_options. In this section There should only be 2 columns options and values. The dataframe will be iterated through and if the value is START_DATE, START_TIME, REPORT_START_DATE, REPORT_START_TIME, END_DATE, or END_TIME then the data will need to be modified by the saved values start_date, start_time, end_date or end_time. If statements will be used. Then replace the section with the dataframe.

Naming the INP file

Network

The first thing we need is the network name.

Banklick_

Plumes

Next we will need to identify the plume I would recommend some kind of plume name instead of a combination of the following:

1. wind direction in degrees
2. pollutant amount
3. pollutant name
4. subcatchment source name
5. source location x value
6. source location y value
7. standard deviation 1
8. standard deviation 2

Taking the variables and assume some things to make this a bit more manageable.

Assumptions:

1. We are only using Cesium.
2. The pollutant amount is the same.
3. We can use the source name for we don't need the to note the X and Y coordinates of the Subcatchment in the name.
4. We are keeping the same standard deviation values to make the plume about the same size.

So we either name the Plume with an auto incrment or I take the assumptions above and have the following

30_degrees_S5

Rainfall

This would be the rain gage and the Start date of the rainfall. We are assuming that the rainfall is being placed in the inp file.

_CVG_20220101

Again we could name the rainfalls that we are planing on using like 2022_high_intensity

Currently we are thinking of adding the rainfall into the INP file. This means that we would only have 1 file for each plume. The INP file needs to be unique. Currently the inp is named the Network_Plume1... How distinct we need to get is going to be the question. This is where I need to look at the file creation of HDF5 files. I should be able to create a table here where I can name the plumes.

The signal should be the same name as the inp_file Name for ease of use.

HDF5 File Creating a Plume Dataset that will give a name to the plume with the dataset that will have all the values. The dataset should include the following:

1. wind direction in degrees
2. pollutant amount
3. subcatchment source name
4. source location x value
5. source location y value
6. standard deviation 1
7. standard deviation 2
8. plume name
9. Rainfall Start
10. Rainfall Gage (If we are only using 1 rainfall gage then this can be eliminated)

Testing

Plumes are being generated and we have created an INP file. The INP file works in SWMM UI. Testing in PySWMM will be the next step.